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ABSTRACT 



This study examined whether parent reports of their child's 
service use are more reliable in a family- focused delivery system than in 
more typical service environments. Data were obtained through the Fort Bragg 
Evaluation Project, a longitudinal assessment of a managed care model for 
delivering mental health services to children and adolescents. Analysis is 
based on about 600 youth, of whom about 60 percent received services through 
the family- focused demonstration program and 40 percent through two 
traditional systems of care. Utilization data were obtained from both the 
youth's primary caretaker and institutional records. Analysis indicated that 
parents in the demonstration and comparison groups tended to report roughly 
equivalent rates of service use as found in their institutional records. 
Also, high percentages of agreement between parent and institutional reports 
were found for all service types, in both service systems. Analysis of 
reliability of reports found higher specificity and lower sensitivity for 
both demonstration and comparison groups. (Contains 19 references and 
endnotes.) (DB) 
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Introduction 

Various efforts to restructure the mental health service 
system for children and adolescents have identified the 
value of a full continuum of services, coordinated by a case 
manager, in which families are encouraged to actively 
participate (Stroul & Friedman, 1986). Adequate 
assessments of the effectiveness of such "systems of care" 
require accurate information on service utilization. Very 
often, utilization data are obtained from a parent of the child 
client. This research examines whether parent reports of 
their child's service use are more reliable in a 
family-focused delivery system than in more typical service 
environments that do not explicitly embrace "system of 
care" principles. 
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' Methods 

Sample 

Data were obtained through the Fort Bragg Evaluation 
Project (FBEP), a longitudinal assessment of a managed 
care model for delivering mental health services to children 
and adolescents (hereafter, the "Demonstration"). The 
present research is based on about 600 youth who remained 
in the study six months after intake to the Evaluation, for 
whom collateral service-related data were collected.! About 
60% of this sample received services through the 
family-focused Demonstration, and 40% through two 
traditional systems of care (hereafter, the "Comparison 
groups"). On average, the sample was about eleven years 
old, about 60% male. Seventy percent were white, 20% 
were African American; and 10% were of other or mixed 
races. About 80% lived in a two-parent household, with an 
annual income between twenty and forty thousand dollars. 
Over 90% of the youth had significant functional 
impairment (CAFAS; Hodges, 1990); over 97% obtained at 
least one clinical diagnosis based on parental responses to 
the Child Assessment Schedule (P-CAS; Hodges, Kline, 
Stem, Cytryn, & McKnew, 1982). 2 

Data Sources/Measures 

Utilization data were obtained from the youth's primary 
caretaker (i.e., usually the biological mother) and 
institutional records (i.e., the management information 
system (MIS) at the Demonstration);3 and the Civilian 
Health and Medical Program of the Uniformed Services 
(CHAMPUS) database for the Comparison groups. The 
following service categories are examined: a) 
hospitalization (HOSP), b) residential treatment (RTC), c) 
outpatient therapy (OUT), and d) medical evaluations 
(MEDEVAL).4 For each category, two measures of 
utilization are analyzed: service use (i.e., "yes" or "no") and 
service dose (i.e., the number of service units received). 

Data are based on reports of use in the first six months after 
intake. Because parents may miss the distinction between 
hospitalization and residential treatment, or between 
outpatient care and medical evaluations, two general 
categories of use are created. "Restrictive" care includes 
hospitalization or RTC; "nonrestrictive" care includes 
outpatient care or medical evaluations. 5 

Analyses 

Identical analyses of reliability were performed on the 
Demonstration and Comparison groups. First, the 
percentage of children who used each service type is 
reported based on each data source. Because parent and 
institutional reports can reveal identical rates of use without 
any association between the reports, proportional 
agreements between the two sources are also provided. 
Kappa coefficients, which indicate whether the agreement is 
better than what would be expected by chance (Cohen, 
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better than what would be expected by chance (Cohen, 

1 960) are shown.6 Agreement can arise because parents 
concur with institutional reports of use (i.e., "sensitivity") or 
nonuse (i.e., "specificity") (Kraemer, 1992). Therefore, data 
are also shown to indicate the nature of agreement between 
parent and institutional reports. Finally, correlations 
between collateral reports of service dose are shown. 

Results 

Table 1 shows that parents in the Demonstration and 
Comparison groups tend to report roughly equivalent rates 
of service use as found in their respective institutional 
records. Exceptions include parental overreporting of 
medical evaluations in the Demonstration and of RTC use 
among the Comparison groups. 

Table 2 shows high percentages of agreement between 
parent and institutional reports for all service types, in both 
service systems. The proportional agreement on medical 
evaluations, though good, is the lowest observed. Most 
notably. Demonstration families indicate much better 
accuracy than the Comparison group with regard to 
nonrestrictive care, specifically for outpatient care. The 
kappas indicate that when chance agreement is eliminated, 
parents in both systems continue to report reliable use of 
restrictive care. Low kappas for nonrestrictive services are 
largely due to the relatively inaccurate reports of medical 
evaluations and the large number of youth who received 
outpatient care. 7 In short, participants in the family-focused 
system of care provide good reports overall, better than their 
counterparts in traditional systems with regard to 
nonrestrictive services, though no better with regard to 
restrictive care. 

Figure 1 shows that reliability depends on whether one 
considers the sensitivity or specificity of reports. 8 
Regarding restrictive services, the strong overall agreement 
at the Demonstration (in Table 21 is largely explained by 
high specificity (98%). Sensitivity of these reports is 
relatively low (69%), with parents at the Demonstration 
tending to underreport.9 The pattern among families in 
traditional systems is similar; they also show greater 
specificity (94%) than sensitivity (86%). While their 
specificity rate is comparable to that of the Demonstration, 
their sensitivity rate is higher. 

The findings with regard to generalized, nonrestrictive care 
show greater sensitivity than specificity in both types of 
systems. Outpatient care in particular is overreported. 
However, this finding must be interpreted in context of the 
number of clients who received outpatient care. When so 
many receive the care, only a few cases incorrectly reporting 
its use would generate a high inaccuracy rate. 1 0 The 
skewness of the sample's distribution is less severe in the 
Comparison group, where 1 1% did not receive outpatient 
care. Indeed, perhaps the most striking finding about reports 
of nonrestrictive services is the low specificity rate (17%) in 
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of nonrestrictive services is the low specificity rate (17%) in 
the traditional systems. Sensitivity of reports of 
nonrestrictive services is comparable in both service 
systems. The low reliability of medical evaluation reports in 
both systems is largely attributable to low sensitivity (i.e., 
underreporting). 

Table 3 shows that parent reports on doses, as with use, are 
more accurate for restrictive than for nonrestrictive care. In 
both systems, correlations between parent and institutional 
records of generalized restrictive care are high and the 
difference between systems is not significant. When 
hospitalization is examined specifically, however, the 
correlation at the Demonstration (r = .88) is significantly 
higher than in the Comparison group (r = .31). On the other 
hand, correlations at the Demonstration for nonrestrictive 
care are low, and significantly lower than among the 
Comparison groups. 

Table 4 summarizes the findings. A plus sign indicates the 
system with the stronger association between parent and 
institutional reports; a minus sign, the system with the 
weaker association. Of the seven specific service categories 
that can be compared, reports for the two service systems 
differed significantly on six. Of these six, four favored the 
Demonstration; two, the Comparison group. The general 
measures indicate that the accuracy of Demonstration 
families' reports of use and dose of restrictive care is about 
the same as observed among families in the traditional 
systems. For nonrestrictive services. Demonstration families 
tend to be more accurate with regard to use, but less 
accurate on questions about the number of service units 
rendered. 

Discussion 

Efforts to provide more effective mental health services for 
children have embraced the value of including children's 
families in the treatment process. This paper explored the 
possibility that parent reports may be more reliable in a 
family-focused system than in traditional systems that do 
not explicitly pursue family-focused interventions. Findings 
suggest that participants in a family-centered system 
generate reliable utilization data. However, their reliability 
does not consistently exceed that observed in traditional 
environments. With regard to restrictive care, this may be 
due to the introduction of innovative, intermediate services 
the Demonstration made available. Families at the 
Demonstration showed a greater likelihood of 
underreporting restrictive care than those in the Comparison 
group, a phenomenon that may relate to the greater number 
of additional services they received through the 
Demonstration. 1 1 Reports of the use of nonrestrictive care 
are good; however, reports of dose are not very reliable, and 
significantly more unreliable than among the Comparison 
groups. 

The Demonstration represented a new and rather complex 
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The Demonstration represented a new and rather complex 
structure with which parents and professionals alike had to 
familiarize themselves. The lack of consistent superiority in 
reliability of utilization reports through the Demonstration 
suggests such organizational challenges may have impeded 
the system's intended capacity to meaningfully include 
parents in treatment planning. Other research (Sonnichsen & 
Heflinger, 1993) has also found that families at the 
Demonstration were not significantly more involved in 
restrictive services than parents at the Comparison sites 
(though they were more involved in nonrestrictive care). 
Together, these findings suggest that we may need to 
reorganize or redouble our efforts to include families as 
active, knowledgeable partners in the therapeutic process. 
Moreover, it may be that any added value of family-focused 
service systems for reliable parent reports may obtain only 
after participants in the system have time to adapt to the 
new organizational environment. 
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Endnotes 

1. The follow-up sample used for this research is more 
limited than in the original Fort Bragg database described 
elsewhere (Bickman, et al., 1995). First, about 10% of the 
Demonstration sample has been excluded because they 
"transitioned" into a different level of care at the time they 
entered the evaluation. These parents may have unusual 
difficulty in distinguishing between services received before 
and after transition, particularly when these services 
represent treatment for the same episode. Second, about 1 00 
cases at the Demonstration, who indicated their childhood 
an initial pretreatment visit but did not return for treatment 
were excluded from analyses. Finally, a small number of 
parents in the Comparison group indicated that outpatient 
and medical evaluations were not paid by CHAMPUS. 

These cases were excluded because the transactions would 
not be included in the CHAMPUS database. Thus, the 
research is based on the approximately 600 cases for which 
service use data could be expected in both the parent and 
institutional database for the same six-month interval. 

2. The child and family profiles of the Demonstration and 
Comparison groups were similar on factors that may 
differentially affect parent recall of services, for 
example&endash;the types or degree of their child's 
problems and their previous experience in the mental health 
system. See Bickman, et al., 1995 for more details on the 
sample and its representation of other youth in treatment. 

3. Previous analysis compared the Demonstration's MIS and 
provider data and found strong agreement between the two 
sources (Bickman, et al., 1995). Thus, the institutional 
record is considered the "gold standard" in this research, to 
which reliability of parent reports is assessed. 

4. The Demonstration offered intermediate levels of care 
(e.g., day treatment) that were unavailable to the 
Comparison groups. Only the four service types offered 
through both service systems are included in this research. 

5. Multiple reports of use of the same service type within 

the six month period are counted once. Dosages associated 
with multiple reports of the same service type are summed. 
Thus, service use and dose represent service types used and 
the total dose received, regardless of the number of episodes | 
or providers associated with each service. | 

I 

6. Kappa estimates of reliability decrease as the proportion \ 
of cases across categories departs fi-om equality. In these i 
data, a sizable majority of cases is classified 
similarly&endash;for example, more than 90% who indicate 
use or nonuse of some services. In this situation, the 
likelihood of chance agreement is extremely high and the 
ability to improve prediction beyond chance (as indicated by 
kappa) is weak. Further, estimates of reliability also 
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decrease as the number of response categories decreases. 
Thus, kappas will be lower for the nominal yes/no 
dichotomies used here to indicate service use then would be 
expected for ordinal measures. Given these properties of 
kappa and of these data, kappas should be interpreted in 
conjunction with other findings presented. 

7. It is difficult to improve upon chance, thus obtain large 
kappas, when the vast majority of cases fall into the same 
category which is the case here with outpatient care. 

8 Less sensitive reports suggest that parents underreport 
service use; less specific reports suggest parents overreport 
use. Cases (indicated by dots for the Demonstration, 
triangles for the Comparison groups) in the upper right 
quadrant of Figure 1 suggest relatively high specificity and 
sensitivity. Symbols that lie below or to the left of this 
quadrant suggest diminishing specificity and sensitivity, 
respectively. 

9. At the Demonstration, too few cases had reports of RTC 
use, and a sensitivity measure was not calculable. Because 
both sensitivity and specificity measures are required for 
charting. Figure 1 does not include an indicator ("dot") of 
RTC use for the Demonstration. 

1 0. The proportion of clients who overreport is based on the 
total number of clients who did not receive that type of care. 
For the Demonstration, only 2% (N = 5) of families did not 
receive outpatient care. Thus, just two clients incorrectly 
reported use yielding a 40% inaccuracy rate. 

11. Summerfelt, Foster, & Saunders, (1966) report greater 
service use among children at the Demonstration than at the 
Comparison sites. Further, auxiliary analyses were 
conducted on Demonstration clients to ascertain whether 
their underreporting of restrictive services was associated 
with overreporting of other service types. Findings showed 
that underreporters tended to report a greater number of 
other services than parents who accurately reported services, 
particularly, intermediate residential services&endash;those 
that parents may be most likely to confuse with traditional 
restrictive settings. And, official data corroborated the 
parents' reports&endash;underreporters did in fact receive a 
somewhat greater number of other services than accurate 
reporters (3.0 vs. 1.5). This suggests that the basis for 
underreporting may be due to confusion that understandably 
may result when multiple services are provided. However, 
confidence in these findings is dampened by the small 
number of clients at the Demonstration who received 
intermediate services. 
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